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Abstract 

During the 2013-14 academic year, Harvard University piloted the use of Massive Open Online 
Courses (MOOCs) as tools for blended learning in select undergraduate and graduate residential 
and online courses. One of these courses, The Ancient Greek Hero, combined for-credit 
(Harvard College and Harvard Extension School) and open online (HarvardX) groups into a 
single online unit, marking the first time a MOOC was used simultaneously by both tuition¬ 
paying, credit-seeking students and non-paying, non-credit students enrolled exclusively online. 
In this article, we analyze and compare the online behavior of students and participants in three 
groups that simultaneously participated in The Ancient Greek Hero via the edX platform. We 
find that, in similar fashion to a traditional learning setting, students enrolled in all three versions 
of the course engaged the online content in a transactional way, spending their time and effort on 
activities and exercises in ways that would optimize their desired outcomes. While user behavior 
was diverse, HarvardX participant engagement tended to be either very deep or virtually 
nonexistent, while College and Extension School students displayed relatively homogenous 
patterns of participation, viewing most of the content but interacting mostly with that which 
affected their overall course grades. Ultimately, educators who intend to utilize MOOC content 
in an effort to apply blended learning techniques to their classrooms should carefully consider 
how best to incorporate each online element into their overall pedagogical strategy, including 
how to incentivize interaction with those elements. Further, for MOOCs to have maximum 
impact, they must address multiple learner motivations and provide participants with multiple 
modes of interaction with the content and with their peers. 
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Introduction 

During the 2013-14 academic year, Harvard University strategically piloted the use of 
massive open online courses (MOOCs) as a primary tool for the implementation of blended 
learning techniques in select undergraduate and graduate courses offered by Harvard College, 
Harvard School of Public Health, and Harvard Extension School. One of these pilots, CB22.1x 
The Ancient Greek Hero (Fall 2013), combined the for-credit and open online groups into one 
instance of the online platform, rather than creating separate instances for each participating 
group to enroll in and utilize, as was done in the other courses. To our knowledge, this pilot was 
the first time that a single instance of an existing or in-production MOOC was used 
simultaneously by both tuition-paying, credit-seeking students and non-paying, non-credit 
students enrolled exclusively online. In this paper, we describe the online behavior of students 
and participants in the three groups that simultaneously participated in CB22.1x. 

Online and blended learning are neither new nor unfamiliar in higher education (Rooney, 
2003; Twigg, 2003; Dziuban, Hartman & Moskal, 2004; Ross & Gage, 2006). However, 
MOOCs are still relatively recent phenomena, and educational institutions remain in the early 
stages of experimenting with their integration into traditional courses (e.g. Bruff, et ah, 2013; 
Bebell, Fernandes & Petersen, 2014; Hollands & Tirthali, 2013, 40-41, 124-125; Fowler, et al., 
2014; Picciano, 2014). Thus far, research suggests that learning outcomes are maximized when 
MOOCs are utilized as supplements to, rather than wholesale replacements for, traditional 
learning experiences. For example, San Jose State saw negative results from their attempt to 
replace traditional remedial and introductory courses with online learning experiences, but has 
experienced more positive results from blended learning experiments in which select MOOC 
content supplements traditional classroom instruction (Ghadiri, et al., 2013). 

Since the initial San Jose State experiment, several more recent studies have been 
produced which evaluate blended learning experiences, in which MOOCs are only one facet of 
content delivery and engagement (Kolowich, 2013a; 2013b; Lewin, 2013; Firmin, et al., 2014; 
Bebell, Fernandes & Petersen, 2014). A wide-ranging experiment at the University System of 
Maryland, for example, resulted in positive academic outcomes while reinforcing the importance 
of the in-person experience (Griffiths, et al., 2014). An experiment in integrating a Stanford 
MOOC on machine learning into a residential course at Vanderbilt University, on the other hand, 
surfaced as a concern the relationship between online and in-person portions of the class (Bruff, 
et al., 2013). These studies and their peers have demonstrated that the strategic incorporation of 
MOOCs into traditional courses can produce a range of academic results. These extend from 
being on par with standard course offerings (Bowen, et al., 2012; Griffiths, et al., 2014; Yousef, 
et al., 2015; Zhao & Ho, 2014) to significantly improving student outcomes in comparison to 
non-blended versions of the same course (Ghadiri, et al., 2013). These data points suggest that 
MOOCs may most effectively serve as a multimedia resource, such as an interactive textbook, 
rather than as a self-contained course in the traditional sense (Bruff, et al., 2013; Emanuel, 2015; 
Fowler, et al., 2014; Means, et al., 2013; Yousef, et al., 2015; Zhao & Breslow, 2013). 

HarvardX, the organization within Harvard University dedicated to producing and 
delivering MOOCs, was founded in 2012. Within the first year of the organization’s existence, 
researchers had already documented common patterns among participants, which have remained 
largely consistent to the present. Participants in HarvardX MOOCs (also referred to as 
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“learners”) tend to have relatively high levels of education, with over half of the participants in 
each course holding a Bachelor’s degree or higher (Ho, et al., 2014). Many sign up for courses, 
but far fewer show broad engagement with the content: most engage with less than half the 
course content and a majority of participants cease activity within the first week of a learning 
experience (ibid). Great diversity exists among the archetypical users (Ho, et al., 2014; DeBoer, 
et al., 2014; Reich, et al., 2014). For instance, users may achieve similar grades, while displaying 
drastically different amounts of time spent on the content. Some users may view all videos and 
engage with all of the readings and assessments, while others may do none of these things but 
have the same overall level of satisfaction with the learning experience (Ho, et al., 2014; Reich, 
et al., 2014). 

This blended learning pilot coincided with the second offering of the Ancient Greek Hero 
MOOC, so the initial iteration (CB22x, Spring 2013) serves as a baseline for comparing 
demographics, activity, and engagement among online learners. Similar to HarvardX users in 
general, CB22x Spring 2013 participants tended to be between 20 and 40 years of age with high 
levels of formal education. They demonstrated diverse patterns of online behavior, although over 
90% engaged with less than half the content, and only 3.2% of registrants obtained a certificate 
of completion (though the certification rate doubles to 5.5% if registrants - those who registered 
but never participated - are removed from the equation) (Reich, et al., 2014, p. 10; Emanuel, 
2015, p.267). 

Following the pilot study, Bergeron (2014) performed a qualitative analysis of five 
courses that used MOOC content in such a way, while Zhao and Ho (2014) utilized a quasi- 
experimental approach to identify the causal effect of a flipped classroom approach on student 
learning in one of those courses, an offering on Chinese history. While Zhao and Ho found no 
substantial difference in outcomes among students in the blended course, both they and Bergeron 
suggest that the nature of this pilot as the first of multiple iterations may mean it represents a 
lower bound with respect to the effect of MOOCs as tools for blended learning on students’ 
academic outcomes. 

This paper does not attempt to evaluate the overall efficacy of blended learning or of 
Harvard’s pilot effort. However, we do intend to contribute additional evidence to the existing 
literature on blended learning, MOOCs, and the nexus between the two, for the dual purpose of 
(1) informing practitioners and (2) providing data for further research in this space. In this study, 
we examine the use of the same MOOC, CB22.1x ‘The Ancient Greek Hero’, across three 
different student subpopulations engaged in three different implementation formats. The first is 
Harvard College undergraduates taking the course Culture and Belief (CB) 22 in residence for 
General Education credit. The second is students at Harvard Extension School (HES) taking the 
writing-intensive course CLAS E-116/w for undergraduate or graduate credit. The third is 
participants in the edX MOOC (known informally as HeroesX) engaging in the project wholly 
online, at no cost and for no formal academic credit. Though all three subgroups were enrolled in 
a single instance of the online content, and interacted together via a single integrated online 
discussion board, Harvard College and HES students were presented with a blended learning 
format that incorporated texts, video, and assessments from the MOOC with in-person or web- 
based class meetings. HeroesX participants engaged the online content in an asynchronous, self- 
paced learning experience. 
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Background 

In the fall semester of 2013, a team from HarvardX and the Derek Bok Center for 
Teaching and Learning at Harvard University’s Faculty of Arts and Sciences partnered with the 
professors and teaching teams from five Harvard courses. The purpose of the collaboration was 
to investigate the use of existing or in-development HarvardX MOOCs as tools for supporting 
blended and activity-based learning techniques in undergraduate and graduate for-credit courses. 
The approach supplemented in-class meetings with online multimedia content, including videos, 
discussion forums, textbooks, and assessments, with the intent of enriching the learning 
experience by increasing the modes by which students interact with the material, and by 
repurposing the class time traditionally used for lecture to allow for different types of 
engagement (Bergeron, 2014). 

The MOOC CB 22. lx ‘The Ancient Greek Hero ’ was utilized in an effort to implement a 
blended learning approach in the Fall 2013 iteration of the long-running residential course of the 
same name. In this approach, students spent considerable time outside of class engaging with 
multimedia content (readings, videos, self-assessments, and online forum discussions), thus 
allowing in-class time to be spent engaging more deeply with the overall themes of the course, 
primarily via a form of Socratic dialogue in which the professor was a participant rather than the 
sole source of infonnation and opinion. 

The Ancient Greek Hero provides a rich case study in the simultaneous use of a common 
platfonn and content repository by three distinct groups of users. For undergraduate students at 
Harvard College (henceforth College), who enrolled in the traditional general education course 
for college credit, and for Harvard Extension School (HES) students, including local and online 
students taking the course for undergraduate or graduate credit, the MOOC platfonn provided the 
required course textbooks in digital, annotatable format, video dialogues building on the readings, 
online forum discussions, and self-assessment exercises. For College students, the online 
material supplemented twice-weekly class meetings. HES students were provided videos of the 
College class meetings and attended a weekly section meeting either in-person or online, 
depending on each student’s physical location (section meetings were omitted from the College 
course in 2013 only, and were reinstated in fall 2014 (Bergeron, 2014)). Both the College and 
HES versions required written assignments in the form of weekly two-to-three sentence micro¬ 
essays and 5 00-word position papers for which select self-assessment questions and discussion 
topics from the HeroesX platform served as prompts. 

The third version of the course consisted of edX registrants (from here, “HeroesX” 
learners) taking the MOOC at no cost and for no academic credit. These participants had access 
to the same online content as the College and HES students, and interacted with members of the 
other groups via the same online discussion board, but they did not have class meetings, sections, 
or other exercises to supplement their experience. Online participants could achieve “Honor 
Code Certificates” in recognition of their level of participation by achieving a cumulative score 
of 50 percent or greater on the online assessments. 

Structurally, the MOOC’s content was divided into 24 chapters, or top-level units of 
content (referred to in CB22.1x as “Hours” for their reflection of the classroom time in an 
academic semester and, aptly, of the units in Homer’s epics). In addition, two opening segments 
were provided, titled “Welcome” and “Introduction” (the latter of which included practice 
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exercises, though they did not count toward final grade or certification), as well as a conclusion, 
for 27 chapters. The entirety of this content was made available to all three groups of learners 
(College, HES, and HeroesX) on the first day of the course, allowing for asynchronous access 
and engagement. College and HES students followed a structured syllabus, with specific 
readings, assignments, and due dates each week. Although this syllabus was presented as an 
“idealized schedule” to MOOC participants who wished to follow along with their for-credit 
counterparts, HeroesX users were both allowed and explicitly encouraged to engage with the 
content at whatever pace, and in whatever depth, was most comfortable to them. 

Demographic Characteristics of Participants 

Harvard College has offered The Ancient Greek Hero for four decades. The MOOC version of 
this course was offered for the first time in spring 2013, with 43,563 students enrolling (Reich, et 
ah, 2014). Enrollment was lower in the fall 2013 second iteration, with 20,598 HeroesX 
participants, and a for-credit population of 237 students from the College and 37 from HES. 
Immediately below, we briefly describe the demographics of the population participating in this 
project. 


Age 

In both iterations of HeroesX, the median age was 29, close to the HarvardX average 
median age of around 28 years. HeroesX users who attained certificates of completion were, on 
average, slightly older than the average age of all HeroesX participants. While HES students 
were similar in age, on average, to HeroesX participants, students enrolled in the College version 
unsurprisingly tended to be younger than their HES and HeroesX counterparts (see Figure 1). 



Figure 1. Mean and median ages among College. HES, and HeroesX students in different 
versions of The Ancient Greek Hero. 


Gender 

Gender balance varies widely between edX courses, although most courses offered by 
HarvardX and MITx to date have tended to have higher proportions of male students. In both the 
spring 2013 and fall 2013 iterations of HeroesX, gender distribution was nearly equal. Slightly 
more male HeroesX students earned certificates in the fall iteration, although the opposite was 
the case in the prior offering (Reich, et ah, 2014). In the College version of the course, male 
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students outnumbered females by a nearly 2 to 1 margin, while females outnumbered males by 
the same proportion in the HES version (see Figure 2). 



Figure 2. Gender distribution of students in College, HES, and HeroesX versions of The Ancient 
Greek Hero, fall 2013 (College n = 237; HES n = 37; edX n = 20,598). 


Education 

Researchers administered a pre-course survey to users in all three groups in an effort to 
shed light on participant information and motivation. In response to this survey, HeroesX 
participants self-reported as having higher overall levels of formal education than both College 
and HES students, a trend also seen in previous MOOCs (Ho, et ah, 2014; see Figure 3). Almost 
two-thirds of HeroesX enrollees had previously earned a Bachelor’s degree or higher. HES 
students had, on average, higher levels of formal education than the College students (over 90% 
of whom, unsurprisingly, had previously earned a high school diploma at most), but less than 
participants enrolled in HeroesX. 



■ Master's or Higher 

■ Bachelor's 
□ High School 

■ Less than High School 


HeroesX 


Figure 3. Highest levels of educational attainment among students in College, HES, and 
HeroesX versions of The Ancient Greek Hero (College n = 237; HES n = 37; edX n = 20,598). 
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Categorization of Participation 

Great heterogeneity has been observed in the behavioral patterns of MOOC participants 
(Kizilcec, Piech & Schneider, 2013; Anderson, et ah, 2014; DeBoer, et ah, 2014; Ho, et ah, 
2014; cf. Yuan & Powell, 2013; Liyanagunawardena, Parslow & Williams, 2014). Because 
students have a variety of personal motivations for enrolling and participating in MOOCs, their 
activity on the edX platform (watching videos, reading texts, completing assessments, 
participating in forum discussions, etc.) can vary widely. Despite the vast number of individual 
patterns of activity, Ho, et al. (2014) employed a simple categorization of participation patterns 
among HarvardX users based on the extent to which users engaged in course content: 

• Registrants, who enroll in the course, but do not engage with any of the content; 

• Viewers, who enroll and engage with less than half of the content; 

• Explorers, who enroll and engage with more than half the content; and 

• Certificate Earners, who enroll and who meet the requirements to obtain certification. 

As Figure 4 shows, each of these categories is made up of a subset of its antecedent. For 
HeroesX participants to qualify for a certificate of completion (officially referred to as an “Honor 
Code Certificate”), they were required to achieve an average of 50% or above on the assessments 
included in the courseware. Scoring in the MOOC was solely dependent on these assessments, 
which consisted of four multiple-choice content questions (worth one point each) and four 
multiple-choice annotation questions based on close readings of primary texts that accompanied 
each of the 24 “Hours.” 


Registrants 



Figure 4. The four categories of MOOC participants. 

In Figure 5, we apply this categorization to the Fall 2013 version of The Ancient Greek 
Hero. The grades of HeroesX, College, and HES participants are plotted against the number of 
total chapters they viewed out of 27, using the above paradigm to describe users based on the 
extent of their course participation. Participants who signed up for HeroesX but viewed no 
chapters are the registrants. The lower left quadrant consists of viewers, who examined less than 
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half the content. The lower right quadrant shows the explorers, who viewed more than half the 
chapters but did not obtain certification. The certificate earners are those in the right upper 
quadrant. 



Number of Chapters Viewed (out of 27) 

HeroesX ♦ HES 
° College 


Figure 5. Scatterplot of grades versus chapters viewed for College, HES, and online students in 
HeroesX (College n = 237; HES n = 37; HeroesX n = 20,598). 

Where do College and HES students fall within this categorization paradigm? The plus 
signs (+) and open circles (o) in Figure 5 denote College and HES students, respectively. 
Overwhelmingly, students in these two versions of the course viewed over half the content, 
making them primarily explorers and certificate earners. Very few in either group viewed less 
than half the chapters, and no students from these versions merely registered (compared to over 
one-quarter of online users who remained only registrants, never engaging the MOOC content at 
all). Interestingly, while explorers comprise the largest percentage of College and HES students, 
the smallest proportion of HeroesX participants - just 1% - fall into this classification, with 
registrants (34%), viewers (62%), and certificate earners (3%) far outnumbering them. This 
suggests that College and HES students kept with the requirements of their respective credit¬ 
bearing courses, as might be expected in a traditional educational enviromnent, compared to the 
nontraditional ways in which MOOC participants engage online content. 

To compare the activity patterns of the for-credit students with the HeroesX participants, 
we conducted a principal components analysis. To measure participation patterns, we include 
participants’ final course grades, the number of chapters and videos viewed, their number of 
forum posts, and the number of days active. The first principal component, which accounts for 
over half of the variance in the participation construct, is primarily characterized by higher 
grades, higher numbers of days active, and more chapters viewed (see loading plot displayed in 
Figure 6). The second component, which accounts for about one-fifth of the variance in the 
participation construct, seems to measure the number of videos watched. 
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Figure 6. Loading plot of principal components 1 and 2, showing that days active, chapters 
viewed, and grades primarily characterize PCA 1, while PCA 2 is primarily characterized by the 
number of videos viewed. 


Using these two components, in Table 1 we show mean principal component scores by 
participant group. The mean scores suggest that HeroesX participants who attain certification 
tend to have the highest number of chapters viewed, the highest grades, and highest numbers of 
days active, while HeroesX Viewers have the lowest. 


Table 1 Mean composites measuring participation based on grades, chapters viewed, videos 
viewed, number offorum posts, and the number of days active 


Group 

PCI 

PC2 

HeroesX Viewers 

-0.444 

0.037 

HeroesX Explorers 

2.976 

0.927 

HeroesX Certified 

5.052 

-0.739 

Harvard Extension 

3.920 

-0.381 

Harvard College 

4.338 

-0.393 


Note. The first principal component explains over half of the variance in the construct. The 
second principal component explains nearly one-fifth of the variance in the construct. The 
remaining three principal components are not shown, as combined, they only account for 
approximately 28% of the variance. 
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Comparing the difference in mean scores, HES and College students appear most like 
each other in their mean scores for both components 1 and 2 (see Table 1 mean component 
scores). When comparing the mean scores of HES and College students to HeroesX participants, 
HES students appear most similar in their participation patterns to HeroesX explorers (diffcertified 
= 1.132 v. diffExpiorers = -944), while College students appear somewhere in between HeroesX 
explorers and certificate earners (diffcertified = 0.714 v. diffExpiorers = 1.362). Throughout the 
remainder of this article, we explore patterns of participation along several dimensions, and 
further compare activity among HES, College, and HeroesX participants. 

Patterns of Participation 

Although students in all three versions of The Ancient Greek Hero used the same edX 
instance, their course requirements, motivations, and performance incentives differed in critical 
ways. As a result, differences in their use of the MOOC platform were expected. College and 
HES students took The Ancient Greek Hero for undergraduate or graduate credit; for the former, 
the course fulfilled a general education requirement, while for HES students, it fulfilled a 
writing-intensive requirement. Only 7 out of 237 College students were concentrating in a 
Humanities-related field, and only one student in the entire class self-identified as a Classics 
concentrator. Because earning college credit is a key motivator, we might expect these students 
to use the MOOC platform in a transactional way, focusing on the content and interactions that 
had the greatest impact on their final grades. 

Conversely, HeroesX participants were not eligible for college credit in exchange for 
successful completion of the MOOC. These enrollees registered and participated for a variety of 
reasons: Around one-fifth of registrants expressed the desire to “engage in lifelong learning,” 
according to the pre-course survey and approximately the same number wanted “to leam from 
the best professors and universities.” Around 13% of HeroesX respondents said at the outset of 
the MOOC that they planned on “completing enough course activities to earn a certificate,” 
while nearly 16% of HeroesX users intended to complete most of all of the readings and 15% 
intended to complete most or all of the assignments. 

Although their credit statuses differed, students in all three versions of CB22.1x likely 
overlapped to some degree in their internal motivations for taking the course. HES and HeroesX 
users were more likely to take the course in addition to full-time jobs or family responsibilities, 
while College students chose the course from among a variety of possible course electives. Thus, 
all participants were to some degree interested in, and motivated by, the content of the course. 
Whether this internal motivation dominated external motivations in driving individuals’ actions 
is a matter of interpretation of the empirical evidence. 

Structure and Participation 

All of the course’s content was housed on the MOOC platform, including videos, self- 
assessments, the shared discussion board, and the two textbooks used in the course, The Ancient 
Greek Hero in 24 Hours (Nagy 2013) and the Sourcebook of Ancient Greek Texts Translated 
into English. As a result, participants from all three groups had to create (free) user accounts to 
access the required material. 

College and HES students’ grades were dependent on their interactions with the 
platform content - specifically, the online assessment questions - but not in the same way that 
HeroesX participants’ scores were. Rather than completing the eight multiple choice questions 
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provided with each Hour of online content, College and HES students were required to write 
brief essays in response to selected questions from those exercises. Each week, these students 
submitted one 500-word “position paper,” which was to be written in response to a pre-selected 
question (or set of questions) from the eight annotation exercise questions included in the two 
Hours that made up that week’s online content. The position papers comprised 39% of College 
students’ final grades, and 50% of HES students’ grades. Additionally, students in both for- 
credit groups were responsible for 22 two-to-three-sentence “micro-essays” over the course of 
the semester (two per week, beginning in the second week of class), which were to be posted on 
the online discussion board on the edX platform. The micro-essays made up 22% of College 
students’ final grades, and the other 50% of HES students’ grades. The students responded by 
writing micro-essays that attended to a stand-alone discussion question provided at the end of 
each hour. For logistical reasons, students were also required to email their papers and micro¬ 
essays to the appropriate members of each course’s teaching staff. As a result of the latter policy, 
students were not held strictly accountable for first posting their micro-essays to the MOOC 
discussion board, and because of this, many decided to forego the extra step of posting to the 
forum altogether. 

Conversely, HeroesX students did not have a writing requirement; to earn certificates of 
completion, they only had to demonstrate their mastery of the concepts and content presented in 
the MOOC by achieving a score of 50% or greater in the online self-assessments. Online 
students’ grades depended exclusively on their performance on the two quizzes that accompanied 
each hour. Users could attempt each assessment question only once, and could receive half or 
full credit (1 or 2 points) for the annotation exercises. 

Given their differences in motivations, incentives, and course requirements, we might 
expect HeroesX users to show different patterns of usage than students in the for-credit classes. 
For example, HeroesX students might engage the multimedia materials to a greater degree, 
particularly with the videos, which were meant to serve not as a centerpiece of the course’s 
content, but as a supplement to, and extension of, the focal point of the learning experience for 
all three groups: the texts. Additionally, in the absence of a traditional class meeting, HeroesX 
participants might engage with the online forums more than College and HES students. 

In Table 2, descriptive statistics for participation and performance are displayed for 
students in each version of The Ancient Greek Hero. As expected, College and HES students’ 
patterns of participation with the edX platform often differed from those of online participants. 
For instance, although the number of course chapters viewed by students in the College and HES 
versions was similar to those in the online version (median of 25 chapters viewed for 
College/HES students compared to a median of 23 chapters viewed for explorers and 27 chapters 
viewed for certified HeroesX participants), students in the former versions often had higher 
numbers of active days compared to students in the online version (median of 51/50 days active 
for College/HES students versus a median of just 17 days active/28 days active for online 
participants who explored/were certified). 
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Table 2 Summary statistics ofparticipation and performance among exclusive categories of user groups 



College 

Extension 

{All 

HeroesX 

Registered Viewed 

Explored 

Certified} 

Days Active ( Mdn) 

51 

50 

2 

1 

2 

17 

28 

Chapters Viewed (Mdn) 

25 

25 

2 

N/A 

2 

23 

27 

Videos viewed (Mdn) 

0 

0 

0 

0 

0 

0 

0 

Videos viewed (Range) 

0 

0-4 

0-236 

0 

0-120 

0-236 

0-62 

Forum Posts (Mdn) 

3 

2 

0 

0 

0 

0 

0 

Forum Posts (M) 

4.9 

6.8 

0.4 

0.01 

0.3 

7.2 

6.5 

% Certified 

24% 

41% 

3% 

0% 

0% 

0% 

100% 

Grade (M) 

31% 

38% 

37% 

0% 

1% 

18% 

75% 

Grade (Mdn) 

25% 

41% 

0% 

0% 

0% 

10% 

75% 

% Total Problems Attempted (Mdn) 

28% 

46% 

0% 

N/A 

0% 

20% 

97% 

% Total Problems Correct (Mdn) 

16% 

29% 

0% 

N/A 

0% 

14% 

65% 

Proportion Problems Correct (Mdn) 

55% 

60% 

0% 

N/A 

0% 

50% 

65% 

Total n 

237 

37 

20,598 

6,971 

12,765 

203 

659 

Percent of Total Participants 

1.1% 

0.2% 


33.4% 

61.2% 

1% 

3.2% 


College and HES students had similar patterns of performance in comparison to online 
explorers and certificate earners according to other measures as well. For instance, College 
students and online explorers had very similar rates of correct assessment questions, with median 
proportions of 16% and 14% total questions correct. Moreover, when considering the proportion 
of problems they correctly answered among those they attempted, College, HES, online 
explorers and online certificate earners had remarkably similar median statistics (ranging from a 
median low of 50% correct among online explorers to a median high of 65% correct among 
online certificate earners, with College and HES students’ medians falling in between. 

Because College and HES students’ institutional grades did not depend on the online 
assessments, and because these students had no requirement to achieve certification in the edX 
learning experience, the distributions of grades and problems attempted differed between these 
groups and HeroesX learners in key ways. In Figure 7, the grade distributions for all HeroesX 
participants for HeroesX explorers and certificate earners and for HES and College students 
appear. In both HeroesX distributions, a noticeable discontinuity exists at the certification cutoff 
of 50% (indicated by the vertical line). This discontinuity could result from “optimizers,” or 
those users who are most incentivized by the certificate as their end goal, who viewed just over 
half the content and completed just enough assessment questions to earn certification. 

The HeroesX grade distribution also includes a spike for users who had perfect or near 
perfect grades. These “completers” may be online users who were most interested in lifelong 
learning (as evidenced by viewing all chapters and completing all assessment questions) and who 
wanted the certificate to demonstrate such learning (as evidenced by the high grade). Neither the 
discontinuity nor the spike of perfect grades appears in the distributions for the College and HES 
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users, who had no incentive to answer the self-assessment questions correctly or to secure an 
online score of above 50%. 


All EdX Online Users 



Online Explorers and Codified Users 



Grade (Out of 100 Points) 


HES 



Grade (Out of 100 Points) 


College 



Note: Distributions include only those with grades greater than 0%. 

Figure 7. Distribution of grades among all HeroesX, HES, and College students in The Ancient 
Greek Hero (All Online Users n = 5,244; HeroesX Explorers and Certified Users n = 786; HES n 
= 32; College n = 228). 

In general, as shown in Figure 8, a positive relationship exists between the number of 
active days on the MOOC site and participants’ final online score (the grade resulting from the 
online self-assessments, not the institutional grade administered to students in the for-credit 
courses). This remains true until around the 75-day mark, at which point increasing the number 
of active days begins to show a negative return (cf. Nesterko & Seaton, 2014). At the same time, 
among HeroesX participants who earned certification, those with very few days active were 
nearly as likely to earn certification as those with many days active, as shown by the relatively 
flat trend line in Figure 8, and by the scatterplot in Figure 9. 


Online Discussion 

Online discussion was a key component of The Ancient Greek Hero, particularly for 
HeroesX users. The HeroesX discussion board provided the primary mechanism for fostering 
discussion among participants from all versions of the course. In fact, promoting conversation 
within and between all three groups was a key factor in the decision to utilize a single instance of 
the MOOC for the two for-credit courses and for HeroesX. To help promote this dialogue, 
College and HES students were explicitly incentivized to initiate online discussions by posting 
each of their 22 “micro-essays,” which counted toward their final grades, to the discussion board. 
However, because of technical difficulties in searching the forums and identifying posting 
histories by user, these “micro-essays” were also emailed to the appropriate teaching fellow(s), 
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thus allowing students to bypass the forums despite the official requirement that they post. 
HeroesX participants, as noted above, had no external incentive for posting to the discussion 
board, as it was not taken into consideration for scoring or certification purposes. However, we 
might expect to see those who were primarily motivated by the opportunity to engage in a 
community learning experience post to the discussion board with some frequency. 



• HeroesX Explorers and Certificate Earners 
° HeroesX Registrants and Viewers 

o HES 

• College 


Figure 8. Grades from edX conditional on the number of days active for HeroesX explorers and 
certificate earners, HeroesX registrants and viewers, College, and HES students (HeroesX 
registrants and viewers n= 4,458; HeroesX Explorer s and Certified Users n= 786; HES n= 32; 
College n = 228). 



• HeroesX Explorers and Certificate Earners + HES 
•> College 


Figure 9. Grades from edX by number of days active for College, HES, and HeroesX 
participants (HeroesX registrants and viewers n = 4,45 8; HeroesX Explorers and Certified Users 
n= 786; HES n= 32; College n= 228). 
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In Figure 10, the distributions of the number of discussion board posts appear for 
HeroesX certificate earners, College, and HES students. The modal number of discussion posts 
was zero for HeroesX certificate earners and HES students, and three posts for College users. 
This, in combination with the fact that the HES and College syllabi required 22 posts each, 
demonstrates the extent to which the option to email “micro-essay” responses directly to the 
teaching staff trumped the requirement to also post these brief responses to the HeroesX 
discussion board. Overall, the College distribution is closer to a normal distribution than either 
the HES or online distributions. 

8 - 

8 - 



5 10 

Number of Forum Posts 


15 

HeroesX Certificate Earners 

■ HES 


| | Colleae 



Figure 10. Bar chart of the number of forum posts by certified HeroesX, HES, and College 
students for all values less than 50 posts (College n = 237; HES n = 36; HeroesX Certified n = 

645). Beyond what appears, 14 HeroesX users posted greater than 50 posts, but these users were 
at the extreme end of the distribution. 

For the most part, except for a small percentage of outliers, students from all groups 
seemed to use the discussion board in a transactional way. Because it was required as part of 
their final grades, College and HES students posted the most on average, though the email 
submission option de-incentivized posting to a sufficient degree that, as noted above, both 
groups’ median and mean posting output was below the amount specifically required in their 
respective syllabi. Interestingly, HeroesX participants, who in the absence of a traditional 
academic community may have stood to benefit most from the discussions, posted the least on 
average. At the same time, although College students post the most on average, HeroesX viewers 
- participants who enrolled and engaged with less than half of the content - actually accounted 
for the majority of users who ever contributed posts to the discussion board, with just over 50% 
of those who posted falling into this category. Around 20% of those who posted were College 
students, and 20% certificate earners. 

Patterns of Assessment Behavior 

Differences in incentives may also account for differences in patterns of quiz taking. In 
the first panel of Figure 11, the distributions of problems attempted appear for three subsets of 
HeroesX participants - those who expressed the intent to earn a certificate in response to the pre- 
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course survey, explorers, and certificate earners. Those participants who indicated that earning a 
certificate was important on the survey did not demonstrate behavior in line with their stated 
intentions, as nearly two-thirds did not attempt any quiz problems. On the other hand, almost 
60% of explorers attempted more than one-tenth of the problems, while over 70% of HeroesX 
participants who eventually earned certification attempted nearly all of the problems. 
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Figure 11. Distribution of the proportion of problems attempted out of total problems for 
HeroesX participants, HES, and College students in The Ancient Greek Hero (HeroesX 
motivated by certification n = 3,395; HeroesX explorers n = 203; HeroesX certificate earners n = 
659; College n = 237; HES n = 37). 


HES and College students also had an incentive to attempt specific self-assessment 
problems. Each week, a selected annotation exercise question (or, in some cases, multiple 
questions) was identified as the prompt for students’ “position paper” essays, and students were 
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informed ahead of time which question(s) that would be. Students could take questions 
selectively, and since attempting a question, (whether the response was right or wrong) would 
display both the correct answer and a paragraph-length explanation from the professor why the 
answer was correct and the other options were incorrect. Because of this, it was beneficial for 
students to at least attempt the self-assessment questions on which each week’s essay was based. 
In Table 3, we provide evidence to support this behavior among College students. While 
HeroesX participants attempted around 30% of questions (essay prompts did not apply to this 
group), College students attempted nearly twice as many questions that served as essay prompts 
as they did non-essay prompt questions. 

Table 3 Comparison ofproportion of questions attempted, grouped by questions that served as 
essay prompts or did not 


Group 

Total Questions 

Essay Questions 

Non-Essay Questions 

HeroesX Viewers 

31% 

25% 

32% 

Harvard Extension 

52% 

57% 

52% 

Harvard College 

39% 

67% 

35% 


Note. Sample includes 2,622 HeroesX participants, 32 HES students, and 229 College students. 

To explore this behavior, we plotted the percentage of essay prompt versus non-essay- 
prompt questions attempted by two groups, College students and HeroesX participants, and 
included a fitted linear function (see Figure 12). The left panel reflects the actual behavior of 
HeroesX participants; we expect the curved functional form given the uneven distribution of 
essay versus non-essay prompts throughout the course. At most, one of eight questions was an 
essay prompt for each hour, although some of the early hours included no essay questions. 
Therefore, conditional on attempting a certain total proportion of questions, more of those 
questions were likely to be non-essay questions than essay questions. Many of the HeroesX 
participants also attempted no questions or attempted all questions. Thus, the overall pattern for 
HeroesX participants, who had no incentive to answer questions selectively, is that they ended up 
attempting more non-essay questions. Conversely, College students attempted substantially more 
essay-prompt questions than they did the remaining 70% of assessment questions, suggesting 
that they did extensively engage in selectively attempting the questions that were most 
incentivized toward their end-of-course grades. 

As noted above, the number of assessment questions engaged by the three broader groups, 
and by the users within the HeroesX population, varied widely, from a median of 28% of total 
questions attempted by College students, to 0% attempted by HeroesX participants overall, to 
97% attempted by HeroesX certificate earners (see Table 3). Interestingly, among those 
questions that were attempted, HeroesX participants’ patterns of success were similar to those of 
College and HES students (see Figure 13). Thus, in comparison to their College and HES 
counterparts, HeroesX certificate earners seem to have achieved certification as a result of 
attempting most or all of the questions, while HES and College students may have failed to 
obtain certification as a result of not attempting as many problems, rather than because they 
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answered the questions they took incorrectly. These patterns reinforce two important points: that 
edX certification signifies the completion of the requirements to a stated standard rather than 
mastery of the content and that failure to obtain certification may not indicate failure to master 
the content and skills of the course. 



Percent of Non-Essay Questions Attempted 


Among College students. 



Percent of Non-Essay Questions Attempted 


Among HeroesX students. 

Figure 12. Proportion of essay prompt questions attempted against non-essay prompt questions 
attempted, for both College and HeroesX students who attempted at least one quiz problem 
(HeroesX n = 2,622; College n = 229). HES plot is not included, but shows relatively 
homogenous scatter along the diagonal. 
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Figure 13. Distribution of proportion of problems correct out of total problems attempted for 
HeroesX participants, HES, and College students in The Ancient Greek Hero (HeroesX 
motivated by certification n = 3,395; HeroesX explorers n = 203; HeroesX certificate earners n = 
659; College n = 237; HES n = 37). 

Conclusion and Implications 

Three key themes emerged from our study of patterns of learner behavior. First, despite 
theoretical aspirations of MOOCs offering both breadth and depth for engaging in the online 
learning space, students in all three versions of the course utilized the edX platform in a 
transactional way, spending their time and effort on activities and exercises in ways that would 
optimize their grades. Second, as cautioned by prior researchers (e.g. Ho, et. ah, 2014), 
certification in the HarvardX space does not imply a specific set of behaviors. It is a binary 
outcome representing a multidimensional concept. While some certified students spend 
significant time engaged in course content, carefully completing quizzes, and engaging with the 
community discussion boards, others seem to engage only with quizzes, over the course of very 
few active hours or days. Given this, we should note that certification in a MOOC does not 
necessarily equate to learning, nor does it necessarily indicate a similar level of content and skill 
mastery as participants in a traditional course (and vice versa for MOOC participants who elect 
not to pursue certification). Likewise, we should be cautious when attempting to compare online 










Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Fonnats 


certification and scores to for-credit course completion and grades. Third, as in many HarvardX 
learning experiences, while user behavior was diverse, HeroesX participants generally trended 
toward one end of the participation spectrum or the other, either registering and viewing less than 
half the content (“ registrants ” and “ viewers ”), or immersing themselves in the majority of the 
content and earning a certificate of completion {certificate earners). College and HES students 
displayed relatively homogenous patterns of participation, viewing most of the content but on 
average, not obtaining the edX certificate of completion. These students produced patterns of 
behavior most similar to a minority of HeroesX participants, which we call explorers (see below 
for further explanation of these terms). 

Similar to a traditional learning setting, students from all three versions of The Ancient 
Greek Hero spent the time and effort necessary to maximize their desired outcomes. In other 
words, both for-credit students and HeroesX participants engaged the online content in ways that 
were best aligned with the incentives they were provided (either internally or externally). Online 
users who achieved certification spent a higher percentage of time attempting problems relative 
to other users, as the online assessments were their sole mechanism for achieving that goal. 
College and HES students, on the other hand, chose to spend time and effort on activities which 
best complemented the offline components of their respective courses, allowing them to 
maximize their own overall performance (cf. Bergeron 2014: 4, 11, 16-18). Other than engaging 
the textbooks and essay prompts made available on the HeroesX platform, this meant spending 
comparatively little time and effort online. 

We draw a few practical conclusions from this analysis. First, educators intending to 
utilize MOOC content in an effort to apply blended learning techniques to their classrooms 
should carefully consider how best to incorporate each online element into their overall 
pedagogical strategy, including how interaction with those elements is to be incentivized (cf. 
Nazarenko, 2015; Cheng, et al., 2017; Sharaj, et ah, 2017). In the case of Culture and Belief 22 
(Harvard College) and CLAS E-116/w (Harvard Extension School), the professor and teaching 
fellows ensured that students interacted with some of the self-assessment questions by basing 
essays on them. This spurred more discussion than might otherwise have taken place by 
requiring use of the forums by for-credit students (though, as noted above, technical limitations 
curtailed this). Some other MOOCs use different tactics to stimulate interaction between 
participants, such as counting forum posts as part of enrollees’ grades. EdX examples of this 
include AmPoX. 1 Poetry in America: The Poetry of Early New England, SW12x China, 
HDS1544.1x Early Christianity: The Letters of Paul, and USW30x Tangible Things: 
Discovering History through Artworks, Artifacts, Scientific Specimens, and the Stuff around You. 
Additionally, one of the courses participating in the fall 2013 pilot, Science of the Physical 
Universe (SPU) 27 Science and Cooking, required students to post their final project to the 
online discussion board. In many other cases, courses forgo this altogether, instead using online 
discussion boards more as a tool of instructor-student communication or for crowdsourcing 
questions and problems; examples of this include, among many others, MCB80x Fundamentals 
of Neuroscience (edX), Understanding Research: An Over-view for Health Professionals 
(Coursera), and Linear Circuits (Coursera). In the case of HeroesX, community engagement and 
discussion was a core component of the overall project’s aims, and one of the chief reasons 
College and HES students were included in the same instance of the MOOC as HeroesX learners 
(Reich, et al., 2014: 19; Emanuel, 2015). In light of this, one aspect of the pilot, which was not 
considered in the present study, is what effect the three learner populations may have had on 
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each other because of their sharing of a single instance of HeroesX. This question may be a 
potential avenue for future research. 

Second, for MOOCs to have maximum impact, they must address multiple learner 
motivations and provide participants with multiple modes of interaction with the content and 
with their peers. As is becoming increasingly emphasized in the literature on MOOCs, 
certification and completion are not the only outcomes that designers and educators should 
consider when designing online learning experiences or when measuring outcomes (Koller, et al., 
2013; Lizilcec, Piech & Schneider, 2013; Stewart, 2013; Yuan & Powell, 2013; Anderson, et ah, 
2014; DeBoer, et ah, 2014; Ho, et al., 2014; Liyanagunawardena, Parslow & Williams, 2014; 
Emanuel, 2015). Researchers and educators must continue to experiment with more flexible 
ways to measure engagement and learning both in MOOCs and in blended, hybrid, and flipped 
credit-bearing courses - particularly those that utilize MOOCs and their content as primary 
resources. In order to do this, practitioners must first gain a better understanding why participants 
are enrolling and participating in MOOCs in the first place, and, insofar as it is possible, think 
about alignment of learner outcomes with their own motivations. This is an area that is rich with 
future research potential, as certain key design concepts may work better in making the content 
of MOOC, online for-credit, and residential courses more useful and engaging to students with 
different sources and levels of motivation. For instance, some MOOCs, particularly in STEM 
fields, allow students to make multiple attempts in order to get the correct answer on quiz 
questions, emphasizing learning the material eventually rather than only the first time (cf. Han, 
Veeramachaneni & O’Reilly, 2013; Kortemeyer, 2014). HarvardX is also relaunching past 
courses with different formats - including The Ancient Greek Hero, which was offered in fall 
2014 as a series of five discrete modules, rather than as a single contiguous learning experience - 
in an effort both to allow learners to better self-direct their experiences, and to gain a better 
understanding of user motivation and engagement. Further studies are possible as more data 
points in this area become available, both within this specific learning experience and across the 
MOOC and blended learning landscape. 

Massive Open Online Courses have the potential to serve both students in blended 
learning courses and learners worldwide who are internally motivated to engage with and leam 
about the content on their own. Practitioners involved in the online learning space are only 
beginning to imagine the possibilities for MOOC design and experimentation to help meet 
learners’ needs. Over time, we can further conduct evidence-based redesigns and reimaginings of 
learning experiences in the pursuit of dynamic learning experiences and improving learning for 
students in all contexts. 
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